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As  budgets  decrease,  it  becomes  increasingly  important 
to  determine  the  most  effective  ways  to  invest  in  modeling 
and  simulation  (M&S).  This  article  discusses  an  approach  to 
comparing  different  M&S  investment  opportunities  using  a 
return  on  investment  (ROI)-like  measure.  The  authors  describe 
methods  to  evaluate  “benefit”  (i.e.,  increased  readiness,  more 
effective  training,  etc.)  received  from  an  investment  and  then 
use  those  metrics  in  a  decision  analysis  framework  to  evaluate 
each  M&S  expenditure.  Finally,  they  conclude  by  discussing  the 
importance  of  viewing  M&S  investments  from  a  Department 
of  Defense  (DoD)  Enterprise  view,  evaluating  investment  over 
multiple  years,  measuring  well-structured  metrics,  and  using 
those  metrics  in  a  systematic  way  to  produce  an  ROI-like  result 
that  DoD  can  use  to  evaluate  and  prioritize  M&S  investments. 
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Successful  Department  of  Defense  (DoD)  Enterprise  modeling 
and  simulation  (M&S)  investment  requires  structure,  persistence, 
and  common  valuation  for  effective  execution.  The  methodology 
summarized  in  this  article  provides  a  systematic  process,  based 
upon  theoretical  aspects  of  capital  structure,  by  \A/hich  DoD  invest¬ 
ments  in  M&S  can  be  compared,  evaluated,  and  directed  to  achieve 
the  greatest  return  on  investment  (ROI)  in  this  “national  critical 
technology”  (House  Resolution  [H.  Res.]  487,  2007). 

To  effectively  apply  a  technology  like  M&S  to  a  DoD  Enterprise, 
application,  or  program,  it  is  critical  to  define  and  assess  rigorous 
measures  of  merit  and  metrics  that  reflect  the  results  of  M&S  appli¬ 
cation  across  the  relevant  spectra  of  management,  mission,  and 
system.  Such  assessments  are  especially  critical  as  budgets  are 
reduced,  opportunities  for  live  tests  and  exercises  are  curtailed, 
and  acquisition  time  lines  are  shortened.  Currently,  most  M&S  value 
assessments  use  metrics  that  are  uneven  in  scope,  very  case-spe¬ 
cific,  do  not  allow/  consistent  aggregation,  or  are  not  \A/ell  structured. 
Additionally,  some  measures  that  are  used,  like  ROI,  are  actually 
incorrectly  defined;  others,  ho\A/ever,  are  undefined,  thus  making  the 
assertions  of  value  at  best  vague,  and  at  \A/orst  incorrect.  Finally,  all 
too  often  important  distinctions  are  not  made  bet\A/een  and  among 
terms  critical  to  consistent  ROI  assessment,  such  as  metrics,  mea¬ 
sure,  scale,  quantity,  quality,  cost,  utility,  and  value. 

Prior  efforts  to  characterize  the  cost-benefits  of  M&S  have 
included  surveys,  assessments,  and  methodological  developments. 
Surveys  summarize  the  results  of  efforts  already  conducted  (Wor¬ 
ley,  Simpson,  Moses,  Aylw/ard,  Bailey,  &  Fish,  1996).  Methodological 
development  articles  provide  insights  into  how/  to  improve  M&S 
value  calculation  (Gordon,  2006).  Assessments  typically  provide 
insights  based  on  one  of  four  approaches:  nominal  description, 
case-based,  business-oriented,  or  multi-attribute  examination. 
All  four  have  advanced  the  state-of-the-art  in  M&S  assessment, 
but  have  not  yielded  an  overall,  rigorous,  and  effective  approach 
for  placing  metrics  in  a  decision  analysis  framew/ork  to  allow/  the 
evaluation  of  M&S  investment.  The  methodology  developed  here 
(Figure  1)  is  distinctive  insofar  as  it  provides  prescriptive  guidance 
w/hile  allow/ing  for  the  comparison  of  alternative  M&S  investments 
(M&S  compared  to  other  M&S  or  M&S  compared  to  other  alterna¬ 
tives  [analysis,  w/ar  games,  etc.])  to  support  a  mission  or  meet  a  goal. 

It  also  facilitates  an  assessment  of  an  M&S  alternative  over  time 
(how/  the  capabilities  provided  change  from  the  initial  application  to 
subsequent  use).  Such  time-considerate  assessments  are  especially 
critical  in  today’s  environment  of  shrinking  budgets.  By  view/ing 
investments  from  a  DoD  Enterprise  view/,  evaluating  investment  over 
a  multiyear  time  line,  measuring  metrics  developed  from  this  view/- 
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FIGURE  1.  M&S  INVESTMENT  METHODOLOGY 
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point,  and  using  these  metrics  in  a  systematic  way  to  produce  an 
ROI-like  result,  the  DoD  can  evaluate  and  prioritize  M&S  investment. 

Market  Context  and  Business  Practice 

stand-alone  strategies  don’t  work  when  your  company’s  suc¬ 
cess  depends  on  the  collective  health  of  the  organizations 
that  influence  the  creation  and  delivery  of  your  product. 
Knowing  what  to  do  requires  understanding  the  ecosystem 
and  your  organization’s  role  in  it.  (lansiti  &  Levien,  2004) 

This  quote  from  the  Harvard  Business  Review  addresses  the 
fundamental  premise  that  commercial  businesses  exist  and  thrive 
(or  not)  within  the  context  of  a  business  environment  much  larger 
than  exists  within  the  boundaries  of  an  individual  firm.  To  succeed, 
individual  firms  must  learn  to  recognize  and  create  value  within 
“the  ecosystem”  in  which  they  exist.  Translated  to  the  domain  of 
DoD  M&S  Enterprise  management,  the  quote,  as  interpreted  by  the 
authors,  could  read: 

Stand-alone  M&S  strategies  don’t  work  when  DoD’s  suc¬ 
cess  depends  on  the  collective  value  created  across  the 
Enterprise,  and  its  creation  and  delivery  of  value  derived 
from  its  investment  in  M&S.  Knowing  what  to  do  requires 
understanding  DoD’s  ecosystem  and  leadership’s  role  in  it. 

Within  the  DoD,  many  organizations  influence  the  creation  of 
value  from  M&S  investment.  On  an  Enterprise  level,  the  key  to  maxi¬ 
mizing  value  is  understanding  who  shoulders  the  costs  and  who 
potentially  derives  value  from  the  allocation  of  resources  to  M&S. 
DoD  investment  strategies  need  to  address,  at  a  minimum,  these 
aspects  of  economic  valuation: 

•  Government  (DoD)  being  the  (only)  buyer  in  many  parts 
of  its  M&S  market  does  discriminate  it  from  private- 
sector  M&S  investment. 
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•  A  lack  of  “marketplace”  from  which  to  gauge  economic 
valuation  often  complicates  DoD’s  efforts  to  make 
sound,  credible  valuation  judgments. 

•  Government  must  account  for  Intangible  benefits  as 
contrasted  to  monetized  benefits  or  simple  revenue. 

•  Unlike  commercial  practice  (e.g.,  corporation-  or  com¬ 
pany-based),  when  the  DoD  invests,  a  misalignment 
often  occurs  between  the  “cost  bearer”  (the  resource 
sponsor)  and  the  “benefit  accruer”  (the  group  that 
gains  an  advantage  from  the  investment),  especially 
when  the  investment  creates  and  returns  value  to  DoD 
components  that  exceeds  the  expected  ROI. 

The  last  bullet  is  particularly  significant.  In  assessing  a  candi¬ 
date  investment,  a  practice  or  methodology  does  not  exist  in  the 
DoD  to  capture  and  characterize  the  future  and  extended  value 
accruing  to  users  beyond  the  primary  recipients  of  the  investment. 
Having  a  methodology  to  capture  such  extended  benefits  could 
change  the  outcome  of  an  investment  decision  from  “not  possible” 
to  “approved,”  and  provide  a  mechanism  for  assessing  all  benefi¬ 
ciaries  for  their  fair  share  of  investment  costs.  Additional  difficulties 
arise  in  the  fact  that  in  many  cases  the  DoD  M&S  investment  cannot 
be  monetized  (translating  elements  of  value  to  units  of  dollars)  in 
a  manner  analogous  to  commercial  business.  Placing  a  monetary 
amount  on  lives  saved,  readiness  improved,  or  warfighters  better 
trained  is  difficult  if  not  impossible.  The  DoD’s  characterization  of 
value  must  often  be  in  terms  that  are  naturally  qualitative,  making 
the  calculation  of  extended  benefit  (analogous  to  the  time-value- 
of-money)  very  different  than  in  the  commercial  sector. 

Across  the  DoD,  the  present  practice  is  to  base  investment  in 
M&S  on  a  number  of  methods;  at  an  Enterprise  level,  however,  the 
practice  is  neither  systematic  nor  consistent.  Writing  in  Acquisition 
Review  Quarteriy,  the  Army  Developmental  Test  Command  Director 
for  Test  and  Technology  C.  David  Brown  and  co-authors  G.  Grant, 
D.  Kotchman,  R.  Reyenga,  and  T.  Szanto  wrote: 

Most  program  managers  justified  their  M&S  investment 
based  on  one  or  more  of  the  following:  reducing  design  cycle 
time;  augmenting  or  replacing  physical  tests;  helping  resolve 
limitations  of  funds,  assets,  or  schedules;  or  providing  insight 
into  issues  that  were  impossible  or  impracticable  to  examine 
in  other  ways.  (Brown  et  al.,  2000) 

Simply  put,  program  managers  (PMs)  are  under  intense  pressure 
to  complete  their  programs  on  budget  and  within  time  lines.  They 
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lack  an  institutional  mandate  to  develop  or  use  M&S  tools  that  may 
have  v\/ider  application  to  other  programs,  or  that  \A/ill  be  cheaper 
to  operate  and  sustain  in  the  long  term  (Bro\A/n  et  al.,  2000).  This 
focus  on  the  program  level,  \A/hile  potentially  good  for  the  PMs, 
can  be  detrimental  to  the  Enterprise  at  large.  When  considering  an 
allocation  of  resources,  PMs  must  consider  not  only  costs,  but  also 
explicitly  definable  benefits.  Equally  important  at  the  Enterprise 
level  are  values  (economies  of  scope),  which  must  be  assigned  by 
leadership  to  complete  the  process  of  estimating  ROI  and  other 
measures  of  value  \A/ith  respect  to  M&S  assets.  The  methodology 
proposed  here  is  a  step  in  accounting  for  these  competing,  yet 
equally  important  value  metrics. 

Stakeholder  and  Community  of 
Practice  Specification 

Understanding  stakeholders  and  their  role-dependent  sensi¬ 
tivities  \A/ithin  the  M&S  community  of  practice  provides  the  context 
\A/ithin  \A/hich  to  determine  M&S  metrics.  DoD  stakeholders  oper¬ 
ate  \A/ithin  a  broad  M&S  market,  \A/here  “market”  includes  the  full 
economic  landscape  over  \A/hich  M&S  products  and  services  have 
impact.  DoD  M&S  stakeholders  fall  into  seven  categories: 

1.  Consumers/Users— End  users  of  M&S-po\A/ered  prod¬ 
ucts  or  of  M&S  services 

2.  Buyers— Expenders  of  funds  for  M&S-pow/ered  products 
or  of  M&S  services 

3.  Sellers— Providers  of  M&S  tools,  data,  or  services 

4.  Investors— Providers/appropriators/deciders  on  expen¬ 
ditures  of  funds  for  M&S  products  or  services 

5.  Approvers/Raters— Providers  of  a  “seal  of  approval”  for 
M&S  tools,  data,  or  services 

6.  Reviewers— Providers  of  “advice  and  consent”  on  M&S 
issues,  including  M&S  products  or  services 

7.  Promoters/Advocates— Independent  providers  of 
“encouragement”  to  the  development  of  the  M&S  mar¬ 
ket  for  M&S-po\A/ered  products  or  services 

Each  stakeholder  category  comes  to  the  M&S  market  with  a 
role-dependent  perspective.  These  perspectives  are  designated 
as:  Program,  Community,  Enterprise,  Federal,  and/or  Society.  For 
DoD  M&S  investment,  the  first  three  perspectives— Program,  Com¬ 
munity,  and  Enterprise— are  considered  to  be  internal  to  the  DoD. 
The  final  two— Federal  and  Society— are  considered  to  be  external 
to  the  DoD.  Stakeholders  provide  another  dimension  that  is  useful 
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in  characterizing  DoD  M&S  investment  considerations  and  eiements 
of  vaiue. 

1.  Program  stakehoiders’  concerns  focus  on  appiicabiiity, 
avaiiabiiity,  and  affordabiiity;  credibiiity,  anaiytic  sound¬ 
ness,  user  friendiiness,  and  entertainment  deiivered, 
as  \A/eii  as  moduiarity,  interoperability,  and  portability; 
and  concentrate  on  systems-of-systems  or  system-level 
functionality. 

2.  Community  stakeholders’  focus  is  on  managing  M&S 
\A/ithin  specific  areas  such  as  acquisition,  analysis,  plan¬ 
ning,  testing,  training,  and  experimentation,  and  is 
oriented  to\A/ard  application-level  indicators  of  success 
or  failure. 

3.  Enterprise  stakeholders’  concerns  focus  on  M&S  capabil¬ 
ities  that  apply  across  diverse  activities  of  the  Services, 
combatant  commands,  and  DoD  agencies. 

4.  Federai  stakeholders’  concerns  focus  on  M&S  develop¬ 
ments  across  departments  and  agencies  of  the  U.S. 
Government. 

5.  Society  stakeholders’  concerns  focus  on  the  role  and 
impact  of  M&S  on  governments,  cultures,  academia, 
industries,  and  populations. 

These  concerns  are  broad  and  encompassing,  and  include  stan¬ 
dards,  policies,  management,  tools,  and  people,  along  \A/ith  reuse, 
interoperability,  collaboration,  interactiveness,  and  sharing  of  assets 
in  a  defense-\A/ide  manner. 


Use  Case 

Developing  and  understanding  use  cases,  including  stakeholder 
needs  and  requirements,  help  determine,  refine,  and  evaluate  the 
process  for  defining  M&S  investment  metrics.  Use  cases  illustrate 
stakeholder  issues  and  role-dependent  sensitivities  together  \A/ith 
investment  decision  processes,  and  serve  to  support  and  guide 
the  definition,  explanation,  and  evaluation  of  processes  and  metric 
alternatives.  We  have  developed  a  frame\A/ork  that  encompasses  a 
consistent  and  complete  set  of  use-case  descriptions  for  use  in  the 
analysis  of  M&S  investment  metrics.  Table  1  lists  the  parameters  of  a 
frame\A/ork  that  provides  a  consistent  and  complete  set  of  use-case 
descriptions  to  help  analyze  M&S  investment  metrics.  The  full  report 
of  the  study  details  three  use  cases  from  different  perspectives 
(AEgis  Technologies  Group,  2008).  The  use  cases  examine  exercise 
options.  Live,  Virtual,  Constructive  (LVC)  middle\A/are  choices,  as 
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TABLE  1.  M&S  USE  CASE  FRAMEWORK 


Parameter 

Selected  Values 

What/Where 

Investment  situation,  investment  goal,  investment 
time  line,  asset  types,  asset  numbers,  other  asset 
information,  geographical  constraints 

Who 

Stakeholder  market  category,  stakeholder 
perspective,  stakeholder  office 

Why 

Concerns,  issues,  forcers,  drivers,  constraints 

When 

Near-term  investments,  mid-term  investments, 
long-term  investments,  schedule  constraints 

How 

Costs  (near  term,  mid  term,  long  term) 

So  What 

Result,  benefit,  utility,  cost  savings 

Data  Support 

Sources,  pedigree,  availability,  timeliness 

well  as  conceptual  nnodellng  alternatives  for  the  Missile  Defense 
Agency.  In  each  of  these,  the  steps  In  this  process  are  delineated 
and  discussed,  sample  data  Included,  and  a  decision  recommended 
based  upon  the  given  scenario.  Due  to  space  limitations,  we  were 
unable  to  Include  them  In  this  article. 

Assets 

To  fully  understand  DoD  M&S  Investment,  It  Is  also  critical  to 
Identify  those  Items  that  DoD  buys.  We  first  define  the  difference 
between  assets  (items  for  DoD  investment)  and  consumables  (in 
accounting  terms:  expenses).  Then  we  list  the  assets  and  catego¬ 
rize  them  depending  upon  the  point  of  view  (POV).  For  example, 
if  one  views  assets  from  the  DoD  POV  (Acquisition,  Analysis,  Plan¬ 
ning,  Training,  Experimentation,  and  Testing),  then  the  assets  are 
categorized  one  way.  Alternatively,  if  the  POV  is  that  of  the  DoD 
Enterprise  Community  (which  from  its  M&S  Vision  Statement  articu¬ 
lates  the  categories  of  Infrastructure,  Policies,  Management,  Tools, 
and  People),  then  the  assets  are  characterized  differently  (AEgis 
Technologies  Group,  2008). 

From  a  DoD  perspective,  an  asset  is  defined  as:  "Something  of 
monetary  value,  owned  by  DoD,  that  has  future  benefit. "  A  consum¬ 
able,  on  the  other  hand,  is:  "Something  capable  of  being  consumed; 
that  may  be  destroyed,  dissipated,  wasted,  or  spent."  Jhe  primary 
difference  is  the  concept  of  future  benefit.  “Future”  in  this  sense  is 
typically  thought  of  as  more  than  12  months  in  the  future.  Exam¬ 
ples  of  DoD  M&S  assets  include:  F-16  simulators,  the  Navy’s  Battle 
Stations  21  simulator,  and  the  online  game  “America’s  Army.”  Con¬ 
sumables,  on  the  other  hand,  are  items  such  as  paper,  pencils,  jet 
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TABLE  2.  ASSETS  LISTING 


Hardware 

Software 

Networks 

Facilities 

People 

Products  & 
Procedures 

Computers 

Models 

Communication 

Lines 

Buildings 

Expertise 

Plans/Policies 

Electronic 

Hardware 

Simulation 

Architecture 

Labs 

Experience 

Standards 

Hardware  in 

the  Loop 

Tools 

(CAD/CAM)* 

Transaction 

Protocols 

Ranges 

Skills/ 

Education 

Analysis  Results 

Mock-ups 

Data/ 

Databases 

Physical 

Models 

Operational 

Knowledge 

Conceptual 

Models 

Spares 

Repositories 

Management 

Processes 

'Computer-Aided  Design/Computer-Aided  Manufacturing 


fuel,  printer  ink— all  typically  used  and  depleted  within  12  months  of 
purchase.  In  light  of  this,  those  types  of  items  that  constitute  DoD 
investment  assets,  using  the  DoD  M&S  Vision  Statement  and  the 
DoD  Communities,  are  shown  in  Table  2. 

By  comparing  this  categorization  with  that  developed  by  cross¬ 
mapping  this  list  with  the  DoD  Communities  (from  both  mission 
and  organizational  views),  and  with  the  DoD  M&S  Vision  Statement 
categories,  we  noted  some  interesting  relationships.  To  start,  every 
listed  asset  correlates  to  more  than  one  major  DoD  Community.  For 
example,  every  DoD  M&S  Community  invests  in  the  Asset  Models. 
While  this  is  not  surprising,  it  shows  that  there  may  be  efficiencies 
gained  by  studying  the  Enterprise  view  and  how  the  DoD  invests 
in  models  since  that  investment  is  widespread.  Also,  the  assets  are 
quite  varied  from  the  tangible  items  to  the  esoteric.  This  means  that 
some  assets  are  easy  to  value,  making  the  determination  of  the  cost 
of  the  investment  relatively  straightforward,  and  some  extremely 
difficult.  Finally,  it  is  difficult  to  place  assets  neatly  into  bins.  All 
assets  cross  functional,  mission,  organizational,  M&S  Community, 
and  DoD  M&S  vision  category  lines,  meaning  an  investment  in  any 
one  of  these  assets  affects  multiple  commands,  agencies,  and  per¬ 
haps  Services.  All  categories  and  sub-categories  invest  in  multiple 
assets.  Because  of  this,  to  be  the  most  effective  and  get  the  highest 
ROI,  investing  in  M&S  needs  to  be  viewed  at  the  Enterprise  level, 
not  at  an  individual  Community  level.  A  true  measure  of  investment 
effectiveness  cannot  be  achieved  unless  one  considers  all  the  costs 
and  benefits. 
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Asset  Costs 

A  decision  to  purchase  or  nnodify  an  M&S  asset  shouid  be  based 
on  the  needs  of  the  custonner(s)  and  the  cost  of  the  purchase  or 
modification.  That  cost  and  associated  decisions  are  best  under¬ 
stood  within  the  context  of  muitiyear  fiscai  caicuiations.  In  iooking 
at  costs  and  the  ROI  of  those  costs,  it  is  important  to  again  acknowl¬ 
edge  that  business  and  government  operate  differently.  If  a  business 
were  to  purchase  an  asset,  the  business  owner  would  likely  evaluate 
the  impact  of  the  asset  on  the  bottom  line:  profit.  The  owner  would 
likely  predict  the  changes  in  profit  and  the  costs  to  purchase  or 
modify  the  asset  over  the  useful  lifetime  of  the  asset,  and  then  com¬ 
pute  (“discount”)  all  those  changes  in  profit  and  asset  costs  back 
to  the  current  year  (today’s)  decision  point.  Different  options,  such 
as  “purchase  asset  A”  or  “modify  asset  B,”  can  be  compared  in  this 
way,  even  if  these  options  have  different  payoff  and  cost  streams 
over  varying  numbers  of  years.  The  comparison  of  the  options  in 
terms  of  current-year  dollars  at  the  time  of  the  decision  gives  a 
standard  metric  that  allows  a  fair  evaluation  of  the  alternatives. 

Government  and  industry  cost  comparisons  differ  in  that  while 
government  generally  does  not  compute  profit,  it  does  compute 
changes  in  expenses.  Additionally,  in  government  the  changes  one 
stakeholder  or  one  PM  makes  can  have  cost  impacts  on  another 
PM,  so  one  PM  can  show  cost  savings  while  others  have  the  burden 
of  increased  costs  because  of  a  change  in  an  asset.  This  shows 
once  again  that  considering  the  Enterprise  perspective  across  all 
impacted  programs  is  essential  to  calculating  an  accurate  and  com¬ 
plete  value  of  M&S  investment. 

Typically,  cost  elements  for  M&S  assets  can  be  grouped  into 
useful  classifications  (Office  of  the  Director,  2007)  for  evaluation  of 
alternatives  through  the  calculation  of  current-year  metrics: 

•  Infrastructure:  standards,  architectures,  networks,  and 
environments 

•  Policies  at  the  Enterprise  level  (including  interoper¬ 
ability  and  reuse) 

•  Management  processes  for  models,  simulations,  and 
data 

•  Tools  in  the  form  of  models,  simulations,  and  authori¬ 
tative  data 

•  People  (including  well-trained  and  experienced  users) 


The  overall  study  illustrated  how  an  increased  level  of  granular¬ 
ity  for  these  classifications  could  be  tailored  to  the  project  and  asset 
particulars,  and  could  be  used  to  facilitate  the  calculation  of  costs 
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by  year  (AEgis  Technologies  Group,  2008).  The  following  example 
illustrates  the  type  of  M&S  alternatives  that  could  be  evaluated  using 
a  cost  element  structure  to  characterize  costs  of  several  alternatives 
over  several  years. 


Using  the  Cost  Element  Structure  to 
Compare  Alternative  M&S  Courses  of  Action 

A  simulation  professional  was  directed  to 
establish  an  annual  experiment  in  Alaska  to  evalu¬ 
ate  capabilities  such  as  the  combat  benefits  of  a 
new  system  for  position  determination  of  friendly 
ground  forces.  The  simulationist  will  need  to  evaluate 
alternative  simulations  for  use  in  this  annual  experi¬ 
ment.  Could  a  different  simulation  be  used  each 
year  depending  on  what  systems  are  being  evalu¬ 
ated,  or  would  it  be  acceptable  and  cheaper  to  use  a 
standard  core  simulation  over  the  next  5  years?  The 
cadre  of  simulation  operators  is  limited  in  Alaska, 
so  the  simulationist  must  also  evaluate  distribution 
of  the  simulation  environment  from  other  locations. 

In  this  first  year,  the  position  determination 
system  may  need  to  be  simulated  or  assumed.  Data¬ 
bases  for  geography  and  other  environmental  factors 
may  need  to  be  purchased  with  requisite  lead  time. 
Connectivity  and  simulation  architecture  costs  will 
have  to  be  evaluated.  The  estimated  cost  of  conduct¬ 
ing  the  experiment,  using  all  live  forces,  would  be  the 
most  costly  option,  and  could  be  used  to  estimate 
cost  avoidance  for  the  other  LVC  options. 

Depending  on  the  alternatives  evaluated,  some 
may  be  more  costly  in  the  current  year  and  cheaper 
in  the  out-years;  while  others  may  be  cheaper  in 
the  current  year  but  with  a  high  stream  of  out-year. 
Hence,  the  cost  comparison  of  the  alternatives  is 
evaluated  based  on  the  sum  of  the  discounted  costs 
across  the  entire  5  years  of  the  experiment. 


Results 
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need  to  be  rigorously  described  in  a  manner  that  accounts  for  both 
qualitative  and  monetary  dimensions.  The  approach  developed 
and  detailed  in  this  section  describes  the  metrics  required  for  such 
analyses,  including  types,  variability,  and  application  particularities. 
The  development  of  such  metrics  is  especially  important  in  M&S, 
\A/here  the  impact  of  investment  and  application  is  not  exclusively 
monetary,  naturally  quantitative,  or  sometimes  even  intuitively  obvi¬ 
ous.  Where  the  \A/ord  “results”  appears,  its  use  reflects  the  outcome 
of  M&S;  includes  both  positive  and  negative;  encompasses  terms  like 
value,  utility,  contribution,  benefit,  and  return;  and  allo\A/s  for  both 
monetary  and  qualitative  effects. 


The  results  calculation  methodology  begins  \A/ith  a  series  of 
assumptions  and  definitions.  It  is  assumed  that  decision  makers  in  a 
governmental  agency  are  rational  actors  \A/ho  seek  to  optimize  rel¬ 
evant  outcomes.  Also,  outcomes  can  be  characterized  using  terms 
that  reflect  the  investment  value  of  alternatives  (meaning,  no  hid¬ 
den  agendas  or  overriding  private  concerns).  The  next  assumption 
is  that  the  metrics  can  be  accurately  quantified  (\A/hether  inherently 
numeric  [like  money]  or  subjectively  assessed).  For  this  effort,  we 
define  three  organizing  principles  or  perspectives  that  can  be  con¬ 
sistently  applied:  Program,  Community,  and  Enterprise.  Next,  it  is 
important  to  understand  the  scope  of  the  results  determination. 
For  instance,  \A/ill  they  be  used  to  compare  alternatives  in  meeting 
a  goal  (M&S  to  M&S  or  M&S  to  other  options),  or  to  the  evolution  of 
an  M&S  capability  over  time?  Next,  in  calculating  results  metrics,  it  is 
important  to  define  the  term  “metric”  in  context  (Table  3).  The  next 
step  of  results  metric  calculation  is  measurement  or  assessment. 
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TABLE  3.  RESULTS  METRICS  CONTEXT 


Relationships 

Example 

First  are  the  classes/categories. 

e.g..  Technical 

Associated  with  each  group  are 
characteristics/terms  describing  features. 

Maintainability,  Design 

Associate  these  with  more 

specific  properties. 

Mean  Time  Between 

Failures, Type 

Decompose  these  into  metrics,  standards 
of  measurement,  like  variables. 

Hours,  Days/Compiled, 
Interpreted 

Metrics  values  are  relative  to  a  scale  (a 
specified  graduated  reference  used  to 
measure)  and  may  be  nominal,  ordinal, 
interval,  or  ratio  in  type. 

1-2-3-4-5-6-7-8-9-10 /C-l 

May  range  from  0  or  no  representation 
to  X,  which  X  represents  a  complete 
implementation  of  the  areas. 

Continuously  for  interval 

and  ratio  data 

Metrics  are  assigned  values,  based  on 
the  features  of  the  M&S  (the  act  of 
measurement)  or  mission  requirement. 

e.g.,  9,  Compiled 

Values  can  be  combined  into  aggregate 

measures  of  merit. 

C  =  2*1,  1  =  1,  Value  =  18 

The  focus  here  is  on  qualitative  or  subjective  judgnnents  that  can  be 
numerically  characterized  and  indices  that  are  naturally  quantitative. 
Finally,  it  is  often  very  important  to  aggregate,  calculate,  or  derive 
an  overall  measure  from  a  decision  theoretic  approach. 

Three  perspectives  apply  \A/ithin  the  DoD  to  the  derivation  of 
relevant  M&S  results  metrics  and  the  calculation  of  their  ROI.  They 
are  the  Program  perspective,  \A/hich  includes  both  M&S  programs 
and  programs  or  activities  that  use  M&S  (Os\A/alt  &  Kasputis,  2006); 
the  Community  perspective,  as  described  in  the  Application  Area 
Descriptions  (Os\A/alt,  2005)  (i.e.,  the  “Surfboard  Chart”);  and  the 
Enterprise  perspective,  as  articulated  in  the  Strategic  Vision  for  DoD 
M&S.  Ackno\A/ledging  these  three  perspectives  is  critical,  since  the 
results  metrics  applicable  to  each  are  different  (Figure  2).  Ho\a/- 
ever,  due  to  space  constraints  and  the  desirability  to  vie\A/  M&S 
investments  from  an  Enterprise  aspect,  only  Enterprise  metrics  are 
summarized  here. 

The  Enterprise  perspective  focuses  on  M&S  capabilities  that 
apply  “across  the  diverse  activities  of  the  Services,  combatant 
commands,  and  agencies”  and  thus  presents  goals  that  are  neces¬ 
sarily  broad  and  encompassing.  They  include  standards,  policies, 
management,  tools,  and  people  that  are  collaborative,  interactive. 
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FIGURE  2.  RESULTS  PERSPECTIVES 

_ Enterprise  (Brain) _ 

Leadership,  Implementation,  Business,  Infrastructure,  System  of  Systems 

Enterprise  metrics  reflect  orchestration  and  management-type 
activities 

_ Communities  (Organs) _ 

Design,  Manufacturing,  Sustainment,  Time  to  Market  Alternatives, 
Complexity,  Sensitivity,  Result  Time  Projection,  Familiarity, 
Comprehensive,  Decision  Time,  Test,  Design,  Augmentation, 
Extrapolation,  Completion  Time  Availability,  Scenario  Variation, 
Experimental,  Retention  Time  Discovery,  Doctrine,  Technology, 

Cycle  Time 

Community  metrics  reflect  more  specific  uses  and  yet  can  Include  both 
enterprise-type  and  program-type  metrics  (when  the  program  crosses 
boundaries  within  a  community  or  between  communities) 

_ Programs  (Blood) _ 

Applicability,  Availability,  Affordability,  Rigorousness,  Engaging, 
Usability,  Creditability,  Technical 

Program  metrics  reflect  the  key  dimensions  of  individuai  M&S  system 
developments  or  M&S  use  within  platform  development  of  programs 


and  sharing  of  assets  in  a  defense-wide  manner  that  includes  other 
“governmental  agencies.  International  partners.  Industry,  and  aca¬ 
demia.”  Metric  categories  for  each  were  derived  previously  (Oswalt 
&  Tyler,  2008).  A  sample  set  of  Leadership  metrics  Is  provided  In 
Table  4. 


ROI  Methods 

In  financial  analysis,  the  concept  of  return  is  critical  and  is  prin¬ 
cipally  used  to  measure  the  change  in  “value”  over  time.  As  such, 
return  is  used  by  the  Financial  Community  to  determine  two  impor¬ 
tant  concepts:  (a)  whether  or  not  the  benefit  of  an  investment  (or 
similar  action)  was  positive  or  negative— this  is  the  “direction”  of 
the  change;  and  (b)  how  positive  or  negative  the  change  was— this 
is  the  “magnitude.”  Financial  analysts  typically  calculate  only  one 
value  by  which  both  direction  and  magnitude  can  be  ascertained. 
The  use  of  a  single  value  is  possible  because  analysts  usually  com¬ 
pare  changes  in  a  single,  same  quantity:  U.S.  dollars.  The  two  most 
common  ways  to  measure  return  are  as  a  percentage  increase  in 
a  holding’s  value  between  two  time  periods.  Return  consists  of  (a) 
the  income  and  the  capital  gains  relative  to  an  investment.  It  is  usu- 
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TABLE  4.  ENTERPRISE  METRIC  SAMPLE 


Enterprise  Perspective 

Sample  Metrics 

Term 

(characteristics) 

Definition 

Quality 

Monetary 

Leadership  (class/category) 

Leadership 

statement  of  vision 

and  associated 
advocacy/support 
of  timely  actions 
needed  for  an 
effective  enterprise 
(property) 

#/currency  of  vision 
and  resulting/ 
supporting  docs 
(metric) 

°  senior  leaders 

adopt  vision  within 
their  (other)  areas 

%  alignment  of 
funding  to  vision 
Savings  from 

reduced  unused  sunk 

costs 

Empowerment 

Developers, 

managers,  users 

that  are  engaged, 
asked,  able  to 
make  significant 
contributions 

#  innovative  ideas 

forwarded  without 

solicitation 

%  M&S  decision 

makers  attending 
key  meetings 

Reduction  in  costs 

to  get  new  M&S 
concepts 

Savings  from 

innovative  M&S  use 

Situational 

Awareness 

Decision  maker’s  and 

users'  understanding 

and  awareness  of 

M&S  standards,  tools, 

etc. 

#  meetings, 
conferences, 
repositories,  Web 
portals 

%  critical  information 
exchanged  among 

communities 

Reduction  in  costs 

to  finding  good  M&S 
information 

Cost  savings 

from  reduction  of 

duplicative  efforts 

Management 

Human  Capital 
Management  for 
recruiting,  assigning, 
career  development 

of  M&S  workforce 

%  M&S  billets  staffed 
with  M&S  qualified 
people 

%  M&S  qualified 
personnel  promoted/ 
retained 

Unnecessary 

training/retraining 

costs 

Cost-effective  M&S 

decisions 

Processes 

Adoption  of  rigorous, 
timely,  and  relevant 

standardization 

and  certification  of 
M&S  policy,  tools, 
workforce,  etc. 

#  promulgated 

processes 

consistently  adopted 
Decreased  product 
(policy,  tool,  etc.) 
generation  time 

Reduced  labor, 
travel,  and  software 

reworks 

Savings  from  error- 

rate  reduction 

ally  quoted  as  a  percentage  (INVESTOPEDIA®,  2010);  and  (b)  as  the 
amount  of  cash  (or  revenue)  generated  from  a  set,  fixed  asset  base, 
expressed  as  a  percentage  of  investment.  Examples  of  this  include 
Return  on  Equity,  Return  on  Assets,  Return  on  Common  Equity,  or 
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FIGURE  3.  DIAGRAM  OF  MADM  PROCESS  FOR  DoD  M&S 
INVESTMENT  ORGANIZED  BY  DoD  COMMUNITIES 


Input 

1  input] 

Input] 

Input 

[input] 

Input! 

[Input 

[Input] 

Input] 

Input 

Input 

Input  1 

ROI 

Metrics 

Other 

Metrics 

ROI 

Metrics 

Other 

Metrics 

ROI 

Metrics 

other 

Metrics 

ROI 

Metrics 

Other 

Metrics 

ROI 

Metrics 

Other 

Metrics 

ROI 

Metrics 

other 

Metrics 

Acquisition 

Anaiysis 

Planning 

Training 

Experimentation 

People 

Return  on  Invested  Capital.  Both  of  these  methods  typically  use 
dollars  as  the  unit  of  measure. 

So  how/  do  we  apply  the  concepts  of  financial  analysis  to  DoD 
M&S  projects?  The  concepts  of  magnitude  and  directionality  men¬ 
tioned  previously  are  essential  to  this  endeavor.  To  make  a  decision 
bet\A/een  a  finite  set  of  options,  a  relative  sense  of  order  is  needed; 
that  is,  to  be  able  to  distinguish  \A/hich  project  is  better  than  the 
others.  Therefore,  \A/hile  we  might  not  assign  a  specific  dollar  value 
to  the  benefit  of  one  choice  over  another,  by  using  directionality 
and  magnitude,  we  can  arrive  at  a  “relative  ranking”  that  \A/ill  let  us 
compare  those  options  among  w/hich  we  are  seeking  to  decide. 
Additionally,  the  notion  of  “internal  consistency”  in  evaluating  dif¬ 
ferent  options  is  vital.  If  we  are  not  able  to  gain  an  absolute  value 
(such  as,  say  83  percent),  but  are  to  rely  on  relative  values  (A  is 
better  than  B,  w/hich  is  better  than  C),  we  must  make  sure  that  we 
are  consistently  applying  the  same  evaluation  criteria  to  all  the 
potential  choices.  The  methodology  for  evaluating  DoD  M&S  invest¬ 
ment  described  in  the  follo\A/ing  discussion  meets  these  criteria  and 
is  completely  consistent  \A/ith  the  manner  in  \A/hich  financial  analysis 
seeks  to  evaluate  return. 

Investment  Decision  Process 

Having  no\A/  determined  metrics  for  the  costs  and  results  asso¬ 
ciated  \A/ith  an  investment,  we  are  in  a  position  to  decide  w/hether 
or  not  to  make  the  investment  using  these  metrics  and  others.  Our 
goal  is  to  employ  a  decision  process  that  takes  into  account  the  data 
gathered,  does  not  rely  upon  chance,  is  fundamentally  simple  to 
explain  and  defend,  and  is  consistent  (\A/ould  give  the  same  ans\A/er 
each  time  \A/ith  the  same  data). 
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Rational  actors,  when  faced  with  a  decision,  will  choose  the 
option  that  maximizes  their  gain  by  some  measure.  In  previous 
sections,  we  presented  methods  to  evaluate  the  costs  and  results 
of  an  M&S  investment;  noted  that  monetization  of  these  metrics 
may  be  difficult,  if  not  impossible,  to  perform  for  the  DoD;  and 
discussed  ROI  methods,  including  key  financial  analysis  elements. 
Given  this  environment  with  its  constraints,  we  developed  a  deci¬ 
sion  process  that  produces  an  ROI-like  quantitative  result  for  use 
in  M&S  investment  evaluation.  We  used  assessed  metrics  as  input 
to  a  Multi-Attribute  Decision  Making  (MADM)  network,  which  has 
the  qualities  of  being  robust,  relatively  explainable,  objective,  con¬ 
sistent,  and  once  established,  can  be  executed  fairly  simply.  MADM 
(Figure  3)  is  not  new  and  has  been  shown  to  work  well  in  structuring 
complex  decisions  involving  a  multidimensional  decision  space.  In 
its  simplest  form,  MADM  is  a  weighted  sum.  The  total  utility  score 
is  calculated  by  multiplying  each  attribute’s  normalized  input  score 
by  its  relative  weighting  (which  would  be  assigned  earlier)  and 
summing  all  the  products.  This  process  is  repeated  at  every  layer. 
While  other  formulae  can  be  employed  to  calculate  a  utility  score, 
the  weighted  linear  method  is  most  often  used  due  to  its  simplicity 
and  transparency  (Tompkins,  2003).  In  this  case,  multilayers  are 
desirable  for  a  few  reasons. 

First,  it  allows  for  the  higher  level  DoD  decision  makers  to  put 
different  emphasis  on  certain  communities  by  assigning  different 
weights  to  each  community.  Additionally,  multilayers  are  desirable 
for  transparency  since  grouping  the  metrics  by  community  makes  it 
easier  to  see  how  certain  measures  impact  the  overall  utility  score. 

It  should  be  noted  that  attributes  measured  should  be  mutually 
exclusive  (no  overlap)  to  prevent  one  attribute  from  influencing 
the  final  score  by  a  higher  amount  than  intended.  Additionally,  the 
weights  are  typically  set  by  a  team  of  subject  matter  experts,  which 
should  consist  of  experts  from  every  area  affected  by  the  deci¬ 
sion  under  consideration,  and  these  weights  should  be  reviewed 
regularly.  Finally,  risk  for  an  investment  can  be  incorporated  in  this 
process  either  as  its  own  category  or  as  a  cost  metric  input  to  the 
framework. 


Conclusions 

By  viewing  investments  from  a  DoD  Enterprise  perspective, 
evaluating  investment  over  a  multiyear  time  line,  measuring  metrics 
developed  from  this  POV,  and  using  these  metrics  in  a  systematic 
way  to  produce  an  ROI-like  result,  the  DoD  can  evaluate  and  pri¬ 
oritize  M&S  investment.  The  process  outlined  in  this  article  meets 
these  criteria  and  is  robust,  consistent,  and  adaptable.  If  followed. 
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the  prescribed  methods  and  guidelines  should  allow  the  DoD,  and 
similar  types  of  organizations,  to  make  M&S  investment  decisions 
that  result  in  an  increased  ROI  when  compared  to  the  current  state. 
An  important  next  step  in  the  development  and  use  of  this  meth¬ 
odology  is  its  application.  Whether  as  an  assessment  technique  for 
a  historical  examination  or  an  approach  for  future  M&S  investment 
analysis,  the  techniques  described  herein  would  provide  rigorous 
and  useful  insights. 
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